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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .1 36(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )E<3 Responsive to communication(s) filed on 19 January 2006 . 
2a)M This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-18 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) E3 Claim(s) 1-18 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
1 1 )□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12)D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)D All b)Q Some * c)Q None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 
THIS IS FINAL OFFICE ACTION TO THE PENDING CLAIMS 1-18 

1. Examiner acknowledges applicant's "response" filed on 19 Jan 2006. 

2. In view of the appeal Brief filed on 9/1/2005, PROSECUTION 

IS HEREBY REOPENED. A new non-final rejection to the pending claims 1-18 is set 
forth below mailed on 10/18/2005. 

3. Examiner acknowledges applicant's Appeal Brief filed on 9/1/2005. 

Drawings 

4. The drawings filed on 2/7/2002 is acceptable for examination; purpose. 

! 

Priority 

5. Applicant's claim for domestic priority under 35 U.S.C. 1 19(e) is acknowledged 
based on the provisional application SI.No. 60/329, 770 filed on 10/16/2001 

6. Examiner notes that applicant filed international application no. PCT/US02/32980 
on 1 0/16/2002 is now published WO 03/034284A1 . 
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Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

7. Claims 1-8,10-18 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Breese et al. [hereafter Breese], US Patent No. 5,987,415 filed on July 
30,1998, published on Nov 16, 1999 in view of Mizokawa, US Patent No.6, 230,111 
filed on Aug 6, 1998, published on May 8, 2001 

8. As to Claims 1,8 12,18 Breese teaches a system which including 'method of 
obtaining information regarding an environment for an individual having preferred 
modalities and engaged in activity using a programmable device' [see Abstract, col 4, 
line 57-62, col 6, line 36-44,col 8, line 8-12, col 10, line 23-27, fig 3], environment for an 
individual corresponds to Breese 's computer user interface that including observing 
user behavior, particularly user behavior caused from emotion and personality state as 
detailed in col 4, line 59-60, further it is noted that Breese specifically teaches a model 
that influence of emotion and personality based on Bayesian network as detailed in 

fig 3, programmable device corresponds to program modules being executed by a 
personal computer because program modules include processes, programs and like as 
detailed in col 6, line 40-42; 'sensing at least one psychomotor behavioral element of 
the activity engaged by the individual' [col 8, line 23-28, line 35-41, col 10, line 34-42], 
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Breese specifically teaches Bayesian net integrate various aspects of emotion and 
personality in a single model for example cognitive response as well as physical 
response based on the metal state, further, it is noted that personality nodes capture or 
senses individual emotional and or personality state as detailed in col 10, line 34-42; 
'psychomotor behavior element of the activity engaged by the individual' 
[col 12, line 49-59, fig 6], psychomotor behavior elements corresponds to loud, angry 
voice tones, calm, quite voice related to emotional state and personality of the user as 
detailed in col 12, line 50-52. It is however, noted that Breese does not specifically 

teach "determining the preferred modalities of the individual'. On the other hand, 

«- 

Mizokawa teaches a system which including 'determining the preferred modalities of the 
individual' [col 6, line 55-60], preferred modalities of the individual corresponds to 
Mizokawa's user's emotions as detailed in col 6, line 58-60. 

It would have been obvious to one of the ordinary skill in the art at the time of 
applicant's invention to incorporate the teachings of Mizokawa into modeling user's 
emotion and personality in a computer user interface of Breese et al. because both 
Breese and Mizokawa are directed to user behavior and emotion model, more 
specifically Breese is directed to computer based capturing emotion and personality 
states that including multistage representation of emotional and personality variables 
[see Abstract, col 5, line 22-28], while Mizokawa is directed to controlling object using 
pseudo-emotions and pseudo-personality generated in the object, more specifically 
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recognizing, updating user commands related to pseudo-personality, pseudo-emotions 
and pattern of behavior [see Abstract, fig 2-3]. 

One of the ordinary skill in the art at the time of applicant's invention to 
incorporate the teachings of Mizokawa into modeling user's emotion and personality in 
a computer user interface of Breese et al. because that would have allowed users of 
Breese to use user evaluation, emotion recognition units to observe various pattern of 
behavior or modality, further allowing to determine pseudo-emotions, predetermined 
relationship between patterns of autonomous behavior, [see Mizokawa: col 2, 
line 15-34], thus improving evaluation of user's intellectual work or desire [col 1, 
line 51-60]. 

9. As to Claim 2, Mizokawa teaches a system which including 'modifying at least 
one modifiable environmental unit to at least partially conform to the preferred 
modalities' [col 6, line 46-54, fig 4]. 

10. As to Claim 3, Breese teaches a system which including 'environment unit is 
modified in real-time 1 [col 17, line 35-43]. 

11. As to Claim 4, Mizokawa teaches a system which including 'storing the sensed 
psychomotor behavioral element in a user history' [col 7, line 32-40, line 51-55]. 
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12. As to Claim 5,13,15, Breese teaches algebraic transforms 1 [col 2, line 38-49], 
Breese specifically teaches Bayesian network model associate with various algebraic 
transforms. On the other hand, Mizokawa teaches a system which including 'sensed 
psychomotor behavioral element is stored 1 [col 7, line 53-59, col 8, line 14-29], 
psychomotor behavioral element corresponds to Mizokawa's user's emotions that 
including obedience, aggressiveness, curiosity, cheerful and like as detailed in col 8, 
line 14-29. 

13. As to Claim 6, the limitations of this claim have been noted in the rejection of 
claim 1 above. In addition, Mizokawa teaches determining preferred modalities includes 
determining a preferred combination of modalities and an ordering of modalities by 
preference thereby further defining a focus of the individual's attention 1 [col 7, line 11- 
17, line 21-24], Mizokawa specifically teaches sound/voice recognition, analyzing 
information on the user's gestures] 

14. As to claim 7, Mizokawa disclosed 'modifying the environmental unit to provide 
content in the environment in the preferred combination of modalities and the order of 
modalities by preference whereby the combination and the order are placed in at least 
one respective co-ordinate group of representational geometry to which attention of the 
individual is drawn, as indicated by the psychomotor behavioral element' [col 6, line 46- 
62]. 
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15. As to claim 8, Breese disclosed 'defining a psychodynamic and a cognitive 
behavioral model using preferred combination modalities and the order of modalities' 
[col 5, line 5-10, line 23-28]; 'modifying at least one environmental unit as a function of 
the psychodynamic behavioral model and the cognitive behavioral model' 

[col 8, line 35-43]. 

16. As to Claim 10, the limitations of this claim have been noted in the rejection of 
claim 1 above. And in addition, Breese teaches 'multi-dimensional and has a plurality of 
modifiable environmental units' [col 10, line 59-67]. * 

17. As to claim 1 1 , Mizokawa disclosed 'preprogramming the device to monitor the 
individual for at least one specific types of psychomotor behavioral elements' [col 4, line 
25-46]; 'communicating an occurrence of the specific type of psychomotor behavioral 
element 1 [col 4, line 39-46]. 

18. As to claim 14, Mizokawa disclosed 'memory device to store sensed 
psychomotor behavioral activity of the individual' [col 7, line 60-64]. 

19. As to claim 16, Mizokawa disclosed 'modalities are calculated while sensing 
psychomotor behavioral activity and concurrently used for modifications to the 
environmental units' [col 10, line 15-24]. 
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20. As to claim 17, Mizokawa disclosed 'sensor includes at least one input device for 
a computer and the modifiable environmental unit includes at least one output device' 
[fig 1, col 4, line 51-59]. 

21. Claims 9 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Breese et al. [hereafter Breese], US Patent No. 5,987,415 filed on Ju 30, 1998, 
published on Nov 16, 1999 in view of Mizokawa, US Patent No.6, 230,111 filed on 
Aug 6, 1998, published on May 8, 2001 

22. It is, noted both Breese, Mizokawa do not teach 'modalities calculation by an 
equation', although Breese teaches modeling a user's emotion and personality 
specifically using Bayesian network inference algorithm [see abstract, col12, line 29-35], 
while Mizokawa teaches control system for controlling object using pseudo-emotions 
and personality specifically calculating using rules or functional equations for pseudo 
emotions levels and personalities [col 10, line 15-18]. Therefore, it would have been 
obvious to one of the ordinary skill in the art at the time of applicant's invention to not 
only sense emotions, but also establish relationship between emotions, environmental 
information to evaluate the behavioral output as suggested by Mizokawa 

[col 9, line 16-21]. 
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Response to Arguments 

23. Applicant's arguments in the response filed on 1/19/2006, with respect to the 
rejection of claims 1-18 have been considered but not persuasive, for examiner's 
response, see discussion below: 

a) At page 8- 9, claim 1, applicant argues that "applicant respectfully requests that 
the examiner identify the specific teachings within the prior art that teach or suggest all 
of the claim limitations, specifically "sensing at least one psychomotor behavioral 
element". 

As to the argument [a] above, as best understood by the examiner, Breese et al 
teaches modeling a user's emotion and personality in a computer user interface, more 
specifically Breese's network having user interface that allows user emotional and 
personality states from the behavior observed both agent network and observer [col 4, 
line 62-67], agent network typically "stochastic model preferably Bayesian network [col 
5, line 5-7]. It is also noted that, Breese specifically teaches Bayesian net integrate 
various aspects of emotion and personality in a single model for example cognitive 
response as well as physical response based on the mental state, further, it is noted 
that personality nodes capture or senses individual emotional and or personality state 
as detailed in col 10, line 34-42, therefore, Breese teaches sensing at least one 
psychomotor behavioral element as detail at col 8, line 23-28, line 35-41 , col # 1 0, 
line 34-42. 
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b) At page 9, claim 1 , applicant argues that Breese in view of Mizokawa fails to at 
least teach, disclose, or suggest determining the "preferred modalities of the individual". 

c) At page 9, claim 1 , applicant argues that applicant's claim of determining the 
preferred modalities of the individual is distinctly different from the teachings of 
Mizokawa 

d) At page 1 1 , claim 1 , applicant argues that Mizokawa does not teach identifying 
preferred modalities of an individual. 

As to the arguments [b-d], Mizokawa is directed to controlling object using 
pseudo-emotions and pseudo-personality generated in the object, more specifically, 
pseudo emotions, and relationship between patterns of autonomous behavior and user 
related, and user-unrelated statuses [col 2, line 16-24], further fig 1 specifically suggests 
each element of predetermined pseudo-personality elements, pseudo-emotions and the 
and relationship. It is noted that Breese does not specifically teach "determining the 
preferred modalities of the individual", although Breese suggests Bayesian net integrate 
various aspects of emotion and personality in a single model, and capturing or sensing 
individual emotional state [Breese: col 10, line 34-42]. On the other hand, Mizokawa 
teaches recognizing user's emotions, more specifically patterns of emotional 
expressions for example joyful, sad, surprise, angry, disgusted and like are captured 
and recognized [see Mizokawa: col 6, line 55-60], further it is noted that Mizokawa 
specifically suggests relationship or mapping between emotions and characteristics of 
sounds/voices and facial expressions [col 6, line 60-67] corresponds to preferred 
modalities of the individual. 
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e) At page 10, claim 1 , applicant argues that "Paragraphs 66-75 further detail 
modalities as claimed. As described in paragraphs 66 and 67, "when grid movement is 
determined the modality summations can be selected to determine if the viewer's 
attention is focused on visual, auditory, kinesthetic or other related cues. Based on the 
results of this equation, the web server can prepare in real time what the next 
presentation and interface should be in order to capture more of the viewer's attention 
by presenting the web content in modalities, which the viewer has nonconsciously 
selected" 

In response to the above-mention argument [e], applicant's interpretation of the 
prior art is noted. However, the claim limitation reads, "determining the preferred 
modalities of the individual". In response to applicant's argument that the references fail 
to show certain features of applicant's invention, it is noted that the features upon which 
applicant relies (i.e., "when grid movement is determined the modality summations can 

be selected to determine if the view's attention is focused on visual, auditory, ) are 

not recited in the rejected claim(s). Although the claims are interpreted in light of the 
specification, limitations from the specification are not read into the Claims. 
See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993) . This is not a 
suggestion of any sort. 

As explained in the Mizokawa's reference, Mizokawa recognizing, evaluating 
users or view's emotions, particularly several patterns of emotions that including "joyful, 
sad, surprise, angry, disgusted, and fearful [col 6, line 58-60], further user's emotions 
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may also be recognized based on sound/voice and facial expressions [col 6, line 46-52], 
it is noted that Mizokawa specifically suggests relationship or mapping between 
emotions and characteristics of sounds/voices and facial expressions [col 6, line 60-67] 
corresponds to preferred modalities of the individual. 

f) At page 1 1 . claims 2-1 1 , Examiner applies above discussed claim 1 arguments 
to the dependent claims 2-1 1 . 

g) At page 12, claim 5, applicant argues "nothing in the cited references suggests 
storing sensed information using linear algebraic transforms. 

As to the above argument [g], Examiner disagree with the applicant because 
Breese specifically teaches various algebraic transforms in computer science area, 
applications that including decision-support system, particularly in emotional signals, 
behavior as detailed in col 2, line 38-49. 

h) At page 12, claim 6, applicant argues that Mizokawa does not teach determining 
"preferred combination of emotions" or an "ordering of emotions by preference". 

As to the above argument [h], as best understood by the examiner, Mizokawa 
specifically teaches user's emotions, and establishing relationship between emotions 
and characteristics of sounds/voices and facial expressions, further categorizing 
emotion expressions of the user into different patterns, further degree of requirement 
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are recognized for example voice or speed of the user's speech and like or loudness 
corresponds to preference as detailed in col 7, line 11-17, line 21-24]. 

i) At page 13, claim 7, applicant argues that Mizokawa teaches a plurality of 
methods for sensing emotions, but does not speak in any manner to determining a 
"preference". 

As to the above argument [I], as explained above, Mizokawa specifically teaches 
user's emotions, and establishing relationship between emotions and characteristics of 
sounds/voices and facial expressions [col 6, line 46-62], further categorizing emotion 
expressions of the user into different patterns, further degree of requirement are 
recognized for example voice or speed of the user's speech and like or loudness 
corresponds to preference as detailed in col 7, line 11-17, line 21-24]. 

j) At page 13, claim 8, applicant argues that Breese cannot teach this claim 
limitation if it does not teach determining preferred modalities. 

As to the above argument [j], as best understood by the examiner, Breese 
suggests developing model that represents behavioral aspects, more specifically 
behavior variables that including emotions/personalities behavioral elements for 
example speech attributes, facial expressions and like as detailed in col 5, line 5-10, 
line 23-28., also it is noted that Psychodynamic is simply based on human behavior 



Application/Control Number: 10/071,731 Page 14 

Art Unit: 2166 

and relationships are shaped by conscious and unconscious influences, while Cognitive 
behavior is a kind of similar to psychodynamic based human behavior example of such 
behavior or personality state for example depression, anxiety disorders, phobias and 
like. 

k) At page 13, claim 9, applicant argues that neither references teaches using the 
specifically claimed formula for calculating the combination and order of modalities. 

As to the above argument [k], as best understood by the examiner, Breese 
teaches Bayesian inference algorithm specifically based on user's emotion and 
personality [see Breese: Abstract, col 12, line 29-35], while Mizokawa suggests 
"pseudo-emotions" for example cheerfulness, joy and like [col 9, line 5-6], further 
Mizokawa also suggests preliminary emotion threshold is correlated with pseudo- 
personality information [col 9, line 42-53], and Mizokawa also teaches calculating the 
threshold of each preliminary emotion are compared with various levels for example 
higher than the preliminary emotion thresholds or lower than the preliminary emotions , 
although it is noted that both Breese, Mizokawa do not show "modalities calculation by 
an equation". 

Therefore, it would have been obvious to one of the ordinary skill in the art at the 
time of applicant's invention to combine the Breese, and Mizokawa's references to 
derive modalities calculation by an equation because Breese directed to stochastic 
models [algorithms] of various levels of emotions/personalities that represents 
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"human behavior", while Mizokawa teaches calculating, comparing pseudo-emotions 
in the "pseudo-personality" thus, establishing relationship between emotins, 
environmental information to evaluate the behavioral output as suggested by 
Mizokawa [col 9, line 16-21]. 

I) At page 14, claim 12, applicant argues that for the reasons identical to those 
given in the defense of claim 1 , Breese in view of Mizokawa, fails to teach, disclose, or 
suggest at least "calculating the individual's preferred modalities", rendering the claim 
nonObvious in view of Breese and Mizokawa. 

As to the above argument [I], examiner applies claim 1 arguments as discussed 

above. 

m) At page 14, claim 13-18, applicant respectfully submits that since claims 13-18 
depend on independent claim 12, claims 13-18 contain all limitations of independent 
claim 12. Since independent claim 12 should be allowed, as argued above, pending 
dependent claims 13-18 should be allowed 

As to the above argument [m], examiner applies independent claim arguments as 
discussed above. 
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Therefore, applicant's remarks are deemed not to be persuasive, and 
claims 1-8,10-18 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Breese et al. US Patent No. 5,987,415 filed in view of Mizokawa, US Patent 
No. 6, 230,111 

Conclusion 
The prior art made of record 

5987415 
6230111 



a. US Patent No. 

b. US Patent No. 



THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 . 1 36(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 
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Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Srirama Channavajjala whose telephone number is 
571-272-4108. The examiner can normally be reached on Monday r Friday from 
8:00 AM to 5:30 PM Eastern Time. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Alam, Hosain, T, can be reached on (571) 272-3978. The fax phone 
numbers for the organization where the application or proceeding is assigned is 
571-273-8300 Information regarding the status of an application may be obtained 
from the Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. 
Should you have questions on access to the Private PAIR system, contact the 
Electronic Business Center (EBC) at 866-217-9197 (toll-free) 

Alam, Hosain, T 
SPE, AU2166. 



sc /fe^ 
Patent Examiner. 
March 17, 2006. 



SfWAMA CHANNAVAJJALA 
PWMARY EXAMINER 



